A Delphi study on collaborative learning in distance education: The faculty perspective

This paper focuses on the factors which influence collaborative learning in distance education. Distance education has been around for many years and the use of collaborative learning techniques in distance education is becoming increasingly popular. Several studies have demonstrated the superiority of collaborative learning over traditional modes of learning and it has been identified as a potential solution to some of the weaknesses of traditional distance education courses. There are a rapidly growing number of technologies in use today and educators and practitioners face an increasingly difficult challenge to successfully implement collaborative learning in distance education; precipitated not only from technical advances but also from wider social and organisational concerns. To the best of our knowledge, this study is the first to investigate the factors that influence collaborative learning in distance education, by eliciting the opinions of an expert panel using a Delphi survey. The aim was to produce an integrated list of the most important implementation factors and to investigate the role technology is perceived to contribute. The findings identified seventeen of the most important factors; these factors cover a range of themes including course rationale and design, instructor characteristics, training, group dynamics, the development of a learning community and technology. The potential of technology however does not seem to be fully realised and newer technologies such as multi-user environments would seem to be of limited use in practice according to the expert panel. Introduction There is an increasing interest in using collaborative learning (CL) techniques in distance education (DE) courses. Collaborative learning emerges through the interaction of individuals with other individuals; knowledge is created though these interactions as individuals ‘exercise, verify, solidify, and improve their mental models through discussion and information sharing’(Leidner and Jarvenpaa, 1995). When students work together collaboratively, they not only learn themselves, but they are also contributing to the development of the group (Salas, Kosarzycki, Burke, Fiore and Stone, 2002). Working in groups is not just a valuable way of learning but also develops the abilities for cooperative work, which is essential in the modern working place. Distance education is a broad term that refers to delivering a curriculum to learners who are not physically present on campus. Recent technological advances, along with changing learner demographics have triggered a significant rise in the popularity of this type of education. Incorporating collaborative activities into modern distance education courses should produce graduates who can work effectively and efficiently with others, while also understanding the role of modern information technologies in collaboration, communication and knowledge creation. As distance education has evolved, it has become inextricably linked to technology (Garrison, 1985) and as such information systems involving communication and information technology have become the underlying core of current and future DE innovations and trends (Lockwood, 2001). Technologies that promote communication and interaction can add value to the learning process by enabling the development of higher-order thinking skills, increased involvement, interest and motivation and overall the attainment of higher learning outcomes (Piccoli, 2001). Although emerging technologies offer a vast range of opportunities for promoting collaboration, distance education programs face challenges that may influence the implementation of these technologies (Beldarrain, 2006). It is also important to make a distinction between ‘collaboratively usable applications and collaborative technology’ (Lipponen and Lallimo, 2004). The use of technology therefore is more than a mere supporting infrastructural component, but at the same time is not the only influencing factor and the organisational processes and human interactions surrounding it are also of crucial importance in the implementation of CL in DE. A number of studies have investigated the factors which are relevant to CL in DE, mostly focusing on specific areas of interest, for example; CL and computer supported groups (Brandon and Hollingshead, 1999); student preferences (Beyth-Marom, Chajut, Roccas and Sagiv, 2003); social interaction (Kreijns, Kirschner and Jochems, 2003); issues with CL in DE (Bernard, Rojo de Rubalcava and St-Pierre, 2000); computer supported collaborative learning (for examples see Beldarrain, 2006; English and Yazdani, 1999; Silverman, 1995); success factors of CMC technologies (Tolmie and Boyle, 2000); system characteristics (Pituch and Lee, 2006) and emerging themes in distance learning (Salas et al., 2002). The literature reviewed indicated that several factors impact CL in DE, although each independent study considered only a limited number of factors. The aim of this research was to produce an integrated list of the most important factors, with the aim of establishing the key issues involved in the implementation of CL in DE. Research Method A Delphi survey was chosen for this study, as it is a data collection method that is designed to elicit and organise the opinions of a panel of experts through iterative, controlled feedback. The framework used was based on non-parametric statistical techniques, as outlined by Schmidt (1997), and aimed to answer the following research questions: RQ1: What are the most important factors that influence the effective use of collaborative learning in distance education? RQ2: What is the perceived role of technology in this form of education? Panel Description Rather than focus on a homogenous group (such as lecturers) a cross section of expertise was sought to ensure that the factors identified considered a range of perspectives and not just the views of a particular group. The panel selected was composed of 18 panellists from three geographical regions: Ireland, USA, and UK. They represented collaborative learning through three distinct groupings; nine programme directors using collaborative learning in distance education courses; five lecturers who are currently using collaborative learning techniques in distance education courses; and four academics with experience in the area of computer supported collaborative learning. Twelve universities were represented: Carnegie Mellon University, (United States); Middlesex University, (United Kingdom); New Jersey Institute of Technology (United States); National University Ireland, Galway (Ireland); Oscail, Dublin City University (Ireland); Penn State University (United States); University College Cork (Ireland); University of Edinburgh (United Kingdom); University of Leicester (United Kingdom); University of Hawaii (United States); University of Limerick (Ireland); and The Exploratorium, San Francisco (United States). Criteria used in selecting the participants were based on their involvement with collaborative learning, distance education and computer supported collaborative learning. The average number of years experience in the area of Collaborative Learning was 7-10 years, with 50% of the panel having over 11 years experience. The panel were also highly qualified in the field of distance education with the average number of years experience 7-10 years and 44% of the panel having over 11 years experience. The average number of CL courses managed or taught was 4-7, as was the average number of DE courses managed or taught. The selected participants are considered to be well informed, leading authorities in their field by their colleagues, supervisors and peers. Overall the panel can be considered highly qualified and well equipped to provide opinions on the factors relating to CL in DE, as qualified by the following section. Forty six experts were invited to partake in the study, based on a preliminary identification process. The invitation provided details of what the study would entail and the expected amount of time that would be needed; the survey took place in June/July 2008. As the commitment was quite intensive (4-6 surveys over a period of 6-8 weeks) the acceptance of eighteen candidates to partake was considered significant and in line with suggestions from the literature; Delphi group size depends on group dynamics rather than statistical power and panels of 10-18 experts are recommended (Okoli and Pawlowski, 2004). During the study two panel members dropped out, leaving a panel of sixteen who completed all stages of the study. Survey Rounds Following the approach used by Kasi et al. (2008) rather than have the panellists participate in a brainstorming session, a list of potential factors was developed by reviewing the literature and extracting those which had been previously identified as important to collaborative learning and distance education. The purpose of the list was to provide the panellists with a structured instrument to begin the Delphi process and contained 28 factors for the initial ratification and discussion (see Appendix for factors discussed in the survey). Detailed instructions of the task involved were provided along with definitions and a glossary of terms relating to the factors. This was to ensure that the panel all had the same understanding of the factors that were being discussed and there was no confusion over semantics. Each participant was asked to select the factors that they deemed to be important to CL in DE and to provide details of any additional factors that they considered to be important. In order to gain a snapshot of the technologies currently in use by the panel they were also sent a short survey and were asked to identify those technologies that they currently used for CL initiatives. The results of the initial survey involved the panel eliminating 9 factors from the original list and adding a further 26. The additional items were reviewed and a new consolidated list of 45 factors was developed. As the target size for the ranking of the factors was around 20 items (Okoli and Pawlowski, 2004) a second survey was required to narrow down the consolidated list. Each panellist was asked to select (but not rank) at least 10 of the most important factors (Schmidt, 1997). The results of this second survey provided a list of the 17 most important factors which influence the successful use of collaborative learning in distance education. The first of the ranking rounds was then sent out. The aim of this phase was to determine the level of consensus on the ranking of the relevant factors. The ranked lists were measured using Kendall’s W coefficient of concordance, as it is recognised as one of the best ways for measuring non-parameter rankings (Okoli and Pawlowski, 2004; Schmidt, 1997). The values of W range from 0 to 1, with 0 indicating no consensus, and 1 indicating perfect consensus. The value of W obtained from this first ranking round was 0.148, which suggested weak agreement on the rankings and thus a second ranking round was necessary. As suggested by Okoli and Pawlowski (2004) the second ranking round was listed in order of the mean ranks obtained in the first round. Each expert was asked to revise their rankings for each item, again asking them to explain their rankings and revisions. The response to the second ranking round indicated that the majority of panellists did not wish to change their opinion, with only four of the panel members revising their rankings. However, a number of additional comments were obtained and Kendall’s W improved to 0.221. At this stage it was decided that further ranking rounds would not be required. The original impetus behind the Delphi method was to seek consensus, as expert consensus was believed more likely to be accurate than an individual forecast. Today, consensus is less important for many investigators; with a useful by-product being crystallisation of reasons for dissensus (Armstrong, 1989). Dissensus, or lack of agreement has been identified as a valid outcome (Armstrong, 1989; Skulmoski, Hartman and Krahn, 2007) as it highlights areas of differences in opinion and enables a deeper understanding of the issues involved. With the diverse perspectives of the panel, and the subjective nature of education, the lack of consensus on the rankings of the factors was considered appropriate. The panel had agreed on the most important factors, the lack of strong consensus was on the priority, or ranking, of these factors. Results This study identified the top 17 of the most important factors from a comprehensive list of 54. The following table provides the results of the final ranking round and outlines the factors in ranked order, along with their mean rank and interquartile range (IQR). Rank Description Mean Rank I Q R 1 Instructional design of the activity, activity structure and assessment needs to promote CL 5.80 7.00 2 Tutor teaching style should encourage involvement and participation 6.20 4.00 3 The development of a learning community should be encouraged and nurtured 6.47 8.00 4 The technology used should be accessible to all participants 6.47 10.0 5 Tutor should assume facilitator role 6.67 7.00 6 Personalised, detailed and quality-controlled feedback on assessment work should be provided 7.20 7.00 7 An appropriate rationale for collaborative learning should be developed 7.47 9.00 8 Tutors should be trained for their role 7.53 8.00 9 Promotive interaction should be encouraged within groups 9.13 4.00 10 Group work should promote positive interdependence 10.00 7.00 11 Learning environment should be user friendly and kept simple 10.47 9.00 12 Prior design of collaborative tasks is essential: i.e. design for learning, then e-moderate for participation 10.60 7.00 13 The development of teamwork skills should be explicitly built into the instructional design 10.93 8.00 14 Technology used should enable multiple means of communication 11.07 8.00 15 There should be lots of opportunity for social communications in the early part of the course 11.27 4.00 16 Technology used should enable asynchronous communication 11.87 8.00 17 Tools should support multiple learning styles 13.47 5.00 Table 1 – Results of Delphi Ranking Round The IQR shows the range of opinion on the ranking of the factor; the higher the IQR the greater the range of opinion. The opinion of the panel was varied and the above table highlights the areas which caused most disagreement on the ranking. For example, the factor ranked number 7 has a high IQR of 9.00. Comments on this factor were varied and tended to either come with a very high or very low ranking. Those who ranked it highly deemed it to be ‘Absolutely critical’ as there is ‘‘No point in doing collaborative work for its own sake – rationale must be linked with learning outcomes & tasks must be relevant/substantial & enable students to acquire deeper understanding / knowledge of course content + experience of teamwork etc’. The reasoning behind its low ranking by some of the panel was that ‘This should be obvious’ or ‘not developed, should be known already’. By exploring these comments it was apparent that the priority given to the factors was based on expert perspectives and individual contexts. This dissensus among the panel is discussed in more detail in the conclusion section. The technology questionnaire sent out with the Delphi study (June/July 2008) provides a snapshot of the technologies in use for CL in DE. This questionnaire obtained details of the percentage of the panel which currently use the technology, along with their perception of its perceived usefulness to CL. The following table provides a summary of the results. T echnology % of Panel using this technology Average Rating VLE / Online Forums / Bulletin Boards 100% Extremely Useful Chat Functions / Synchronous Discussion 94% Moderately Useful Computer / Audio Conferencing / VoIP (e.g. Elluminate, Skype) 83% Moderately Useful Collaborative document tools (e.g. Google docs, Word comment) 78% Extremely Useful Email / Email List Server 72% Moderately Useful Wiki Spaces 67% Moderately Useful Social Networking Software (e.g. Ning, del.icio.us, wiki, facebook) 61% Limited Usefulness Blogs 56% Moderately Useful Calendars, Agendas or Schedules 56% Moderately Useful Voting 50% Limited Usefulness Multi-User Virtual Environments (e.g. Second Life) 44% Limited Usefulness Podcasting 44% Moderately Useful Group conferencing (with synchronous audio / video) / Video Conferencing 40% Limited – Moderately Useful Table 2 – Results of Technology Snapshot questionnaire Discussion of F indings Course rationale and design In line with the literature (Brandon and Hollingshead, 1999; English and Yazdani, 1999; Kennedy and Duffy, 2004; Tolmie and Boyle, 2000) course rationale and design is considered highly important. Personalised, detailed and quality-controlled feedback on assessment work should be provided as it is also seen as of high importance. The literature also suggests that the rationale behind the use of Computer Mediated Communication (CMC) technologies is considered important (Tolmie and Boyle, 2000): however this study did not find it to be a high priority, perhaps because in DE it is necessary to use CMC technologies. While there are a number of suggestions around course content discussed in the literature (Bernard et al., 2000; Brandon and Hollingshead, 1999; Silverman, 1995), course subject matter was not found to be particularly important by this panel of experts. Instructor Characteristics The role of the tutor, or instructor, is significant to CL, with ‘teaching style’ considered to be a ‘most important influence’ on involvement and participation (Salas et al., 2002). The expert panel would seem to agree with this as one of the highest-ranking factors directly related to instructor teaching style, suggesting that it should ‘encourage involvement and participation’. Learner-centred courses require the instructor to assume a facilitator role, and again this is suggested as important in the literature (Bernard et al., 2000; English and Yazdani, 1999; Silverman, 1995). The panel concurs with this agreeing that the tutor should assume a facilitator role, and also receive training for the role. Support, both technical and institutional, is not deemed to be as high a priority and did not make it to the final list of factors. Learning Community It would also seem that, in line with the literature (Hiltz, 1998; Kreijns et al., 2003), the encouragement and development of a learning community is considered highly important to the effective use of CL in DE. While there should be opportunity for social communications in the early part of the course, this is not viewed as high a priority. The importance of the group to collaborative learning has been regularly discussed (for examples see (Brandon and Hollingshead, 1999; English and Yazdani, 1999; Hiltz, 1988; Kreijns et al., 2003; Tolmie and Boyle, 2000) and the results of this study would seem to concur, at least with regard to the importance of promotive interaction and the fact that group work should promote positive interdependence. However, other aspects deemed important in the literature, for example, group size (Kreijns et al., 2003; Tolmie and Boyle, 2000) were not supported by this panel. This may be due to fact that the technology available today allows large groups to work quite well. As one panellist pointed out ‘perhaps bigger groups enable the work to be sustained when some of the group are inactive for long periods’. Group interpersonal skills (Kreijns et al., 2003) were also not deemed to be of major importance, while individual accountability (Brandon and Hollingshead, 1999) was seen as important but not ‘most important’ and did not go through to the final list. Student Characteristics Student characteristics are regarded as important in the literature with Bernard et al. (2000) suggesting that that ideally ‘developing a profile of the learner’s knowledge, skills and experience, as well as their perceived needs’ will aid in the design and implementation of effective DE courses. Learner differences involve both the way that students will interact with the technology as well as affect the degree to which they will participate in online collaboration activities (Salas et al., 2002). However, the panel did not concur with the literature; they did not consider student characteristics an important factor in the effective use of CL in DE. It was agreed that while students may have a preference for certain learning styles these can be overcome and adapting learning styles to suit course requirements is part of the learning experience itself. Role of Technology Five technology based factors are among the most important factors. Accessibility to the technology has been highlighted as important in the literature (Bernard et al., 2000) and this has been upheld by the panel, placing this factor in the top five. The lower ranking of the technology factors would seem to be due to the fact that it is deemed a supporting role and less critical than good design and tutor characteristics. While not considering the technology unimportant, the point was made that: ‘with a good plan, and buy-in from teachers, the technology should not matter. Of course there is a need to match the technology to the task – but perhaps we are getting to the position that we are doing this in what seems like an intuitive and natural way. The technologies are (relatively) mature and powerful, so we can achieve our ends with a range of different technologies’. While the above comment recognises that the technologies are now relatively mature and powerful, it would seem that technology is not being fully utilised. One factor that had a wide discordance concerned the ability of technology to cater for different learning styles. While the majority of the panel did not consider this to be a high priority, one panel member fully supported this and felt that the rest of the panel was overlooking it. ‘Tools need to reflect the multiple styles for learning and not assume that students should adapt to purely linguistic ones. This doesn’t mean we have to test and understand every student’s primary learning style, only that we have to design environments that appeal to multiple styles in a variety of ways’. Alavi and Leidner (2001) point out that ‘the role of IT in enabling individualized learning methods, while not new, has received strikingly little attention’; this study supports this view and further indicates that the potential of IT has still not been recognised in practice. The potential of technology to provide an individualised, effective learning environment is not recognised or utilised by the majority of this panel of experts. A recent study (Menchaca and Bekele, 2008) identified that ‘the availability of multiple tools added flexibility to the learning environment’ which helped ensure a successful DE programme, as did the use of technology tools that appeal to multiple learning styles. Perhaps as more research identifies the usefulness of these technological tools they will be utilised more fully in practice. Perceived usefulness of CL technology It would seem that virtual learning environments, including online forums and bulletin boards are of most use to collaborative learning in distance education. The entire panel uses this technology and it received a high level of support on its usefulness. Collaborative document tools are also considered ‘extremely useful’ by the panel, with 78% of them using this technology. The majority of the technology is considered ‘moderately useful’ including audio conferencing and email, even though these had a high percentage of use. It is interesting to note that some of the newer technologies such as multi-user virtual environments and group conferencing appear to have limited usefulness in practice, although recent research into these technologies show a promising outlook (Ketelhut, Nelson, Clarke and Dede, 2010; Salmon, 2009). Conclusion This paper has explored the factors which are deemed to be important to the implementation of CL initiatives in DE courses. Its findings highlight the most important factors which should be considered along with providing a snapshot of the technologies involved. It would seem that in practice, there is suboptimal use of technology in this educational environment. In particular, newer technologies such as multi-user environments, group conferencing and social networks are perceived to be of limited usefulness. These technologies have the potential to enable collaborative learning to take place over distance and as such their perceived lack of usefulness is of concern. If these technologies are to be fully optimised as an enabling factor in collaborative distance education then their educational benefits need to be more strongly highlighted to practitioners. Dissensus on the rankings of the factors While the panel did agree on the most important factors, there was only weak agreement on the rankings of these factors. This lack of agreement highlights the diverse views and opinions of the experts. It would seem that opinions and rankings are based on the context in which the collaborative learning is taking place and the perspectives of the individual experts. Some considered certain factors to be ‘obvious’ and thus not worth ranking highly while others took the opposite approach and ranked the obvious ones as high priorities. Another possibility, for the dissensus, may be that the extensive experience of the panel led to them being less likely to change their opinions based on others views; this may also be why the new technologies have received little support – those who are running successful CL programmes may see little reason to change from the technology and/or practices which they are currently using. However, the purpose of this research was to gain an understanding of the factors from diverse perspectives and as such the lack of strong consensus on the ranking is appropriate. Dissensus is a valid Delphi outcome as it can provide an understanding of the varied opinions on the factors being discussed. The study has highlighted that the use of collaborative learning in distance education is likely to be context specific and while certain factors are always important, the priority of this importance is not necessarily consistent. Future research in specific contexts is encouraged by the authors to explore this further. Implications for IS Research While the study established that technology is among the most important factors, it also highlighted the fact that it is viewed more as a supporting infrastructure than actually adding value to the collaborative learning in more innovative ways. In particular, the use of technology to support multiple learning methods is an area that is not currently optimising the potential of technology. Further research into the role of technology in collaborative learning might consider how the technology is being used and why it is not being fully utilised in practice. The perceived usefulness of the technologies could also be further explored, in particular the lack of support for some of the newer technologies. The fact that Multi-User Environments are used by less than 50% of this panel, along with the suggestion that it is of limited use warrants further research. As these new technologies can actually enable collaborative learning to take place, rather than just support it, it is important to establish why they are not being considered particularly useful in practice. Comparison of the technologies used with the factors identified was not an objective of this study. Further research could investigate how specific technologies are used to support the factors identified in order to understand how the technologies in use are judged to support the important collaboration factors. Also, as an exploratory study, this research did not attempt to determine how these factors influence CL in DE; further research could aim to establish how these factors affect the success of CL in DE. As with any Delphi study, the results are based on a limited number of subjects. While these subjects were chosen following rigorous guidelines, one must be cautious in generalising. The sample is relatively diverse in terms of perspectives on CL in DE and this diversity may have influenced the lack of consensus on the priority of the items. Further research could be carried out with panels of similar perspectives to determine if the results would hold. Implications for practitionersThe panel of experts utilised in this Delphi study are highly experienced and their combinedknowledge has provided a useful guide to the most important factors in this growing area ofeducation. These factors can be grouped into specific areas course rationale and design;instructor characteristics; learning community; assessment; support and training; groupdynamics and technology – to provide practitioners with a focused approach to considerwhen undertaking this type of project.Overall the study provides a practical guide for those considering implementing collaborativelearning in distance education, along with some motivation for future research for the IScommunity regarding the suboptimal utilisation of technology in practice. ReferencesAlavi, M. and Leidner, D. E. (2001) Research commentary: Technology-mediated learning A callfor greater depth and breadth of research, Information Systems Research, 12 (1), pp. 1-10. Armstrong, M. (1989) The Delphi Technique, Princeton Economic Institute, NJ. Beldarrain, Y. (2006) Distance Education Trends: Integrating new technologies to foster studentinteraction and collaboration, Distance Education, 27 (2), pp. 139-153. Bernard, R. M., Rojo de Rubalcava, B. and St-Pierre, D. (2000) Collaborative online distancelearning: Issues for future practice and research, Distance Education, 21 (2), pp. 17. Beyth-Marom, R., Chajut, E., Roccas, S. and Sagiv, L. (2003) Internet-assisted versus traditionaldistance learning environments: factors affecting students' preferences, Computers &Education, 41 (1), pp. 65-76. Brandon, D. P. and Hollingshead, A. B. (1999) Collaborative learning and computer-supportedgroups, Communication Education, 48 (2), pp. 109-126. English, S. and Yazdani, M. (1999) Computer-supported cooperative learning in a Virtual University,Journal of Computer Assisted Learning, 15 (1), pp. 11. Garrison, D. R. (1985) Three Generations of Technological Innovations in Distance Education,Distance Education, 6 (2), pp. 235-242. Hiltz, S. R. (1988) Collaborative learning in a virtual classroom: highlights of findings, Proceedingsof the 1988 ACM conference on Computer-supported cooperative work ACM, Portland,Oregon, United States, pp. 282-290. Hiltz, S. R. (1998) Collaborative Learning in Asynchronous Learning Networks: Building LearningCommunities, WebNet 98 World Conference of the WWW, Internet, and IntranetProceedings, Orlando, Florida, November 7-12, 1998. Kasi, V., Keil, M., Mathiassen, L. and Peterson, K. (2008) The post mortem paradox: a Delphi studyof IT specialist perceptions, European Journal of Information Systems, 17 (1), pp. 62-78. Kennedy, D. and Duffy, T. (2004) Collaboration a key principle in distance education, OpenLearning: The Journal of Open and Distance Learning, 19 (2), pp. 9. Ketelhut, D. J., Nelson, B. C., Clarke, J. and Dede, C. (2010) A multi-user virtual environment forbuilding and assessing higher order inquiry skills in science, British Journal of EducationalTechnology, 41 (1), pp. 56-68. Kreijns, K., Kirschner, P. A. and Jochems, W. (2003) Identifying the pitfalls for social interaction incomputer-supported collaborative learning environments: a review of the research,Computers in Human Behavior, 19 (3), pp. 335-353. Leidner, D. E. and Jarvenpaa, S. L. (1995) The use of Information Technology to enhanceManagement School Education A Theoretical View, Mis Quarterly, 19 (3), pp. 265-291. Lipponen, L. and Lallimo, J. (2004) Assessing applications for collaboration: from collaborativelyusable applications to collaborative technology, British Journal of Educational Technology,35 (4), pp. 433-442. Lockwood, F. (2001) Innovation in distributed learning: creating the environment, In Innovation inOpen and Distance Learning Successful Development of Online and Web-basedLearningKogan Page Limited, London, pp. 14. Menchaca, M. P. and Bekele, T. A. (2008) Learner and instructor identified success factors indistance education, Distance Education, 29 (3), pp. 231-252. Okoli, C. and Pawlowski, S. (2004) The Delphi method as a research tool: an example, designconsiderations and applications, Information & Management, 42 (1), pp. 15-29. Piccoli, G. (2001) Web-based Virtual Learning Environments: A research framework and apreliminary assessment of effectiveness in basic IT skills training, Mis Quarterly, 25 (4), pp.25. Pituch, K. A. and Lee, Y. (2006) The influence of system characteristics on e-learning use, Computers& Education, 47 (2), pp. 222-244. Salas, E., Kosarzycki, M. P., Burke, C. S., Fiore, S. M. and Stone, D. L. (2002) Emerging themes indistance learning research and practice: some food for thought, International Journal ofManagement Reviews, 4 (2), pp. 135-153. Salmon, G. (2009) The future for (second) life and learning, British Journal of EducationalTechnology, 40 (3), pp. 526-538. Schmidt, R. (1997) Managing Delphi Surveys using NonParametric Statistical Techniques, DecisionSciences, 28 (3), pp. 11. Silverman, B. G. (1995) Computer Supported Collaborative Learning (CSCL), Computers &Education, 25 (3), pp. 10. Skulmoski, G., Hartman, F. T. and Krahn, J. (2007) The Delphi Method for Graduate Research,Journal of Information Technology Education, 6 (1), pp. 21. Tolmie, A. and Boyle, J. (2000) Factors influencing the success of computer mediated communication(CMC) environments in university teaching: a review and case study, Computers &Education, 34 (2), pp. 119-140. Appendix – Consolidated L ist of Factors Factor Factor DescriptionInitial List Validated Final F1Students should have prior experienceof collaboration technology√ F2Promotive interaction should beencouraged within groups√√√ F3Students learning style should beconducive to group-work√F4 Tutor should assume facilitator role √√√ F5Tutor teaching style should encourageinvolvement and participation√√√ F6Group members should have adequateinterpersonal skills√ F7Technology used should enablesynchronous communication e.g. MSNchat and teleconferencing√ F8An appropriate rationale for collaboratelearning should be developed√√√ F9Group work should promote individualaccountability√√ F10Course content should encourageinteraction with both tutor and peers √√ F11Tutors should have institutional supportfor their role√√ F12The technology used should beaccessible to all participants√√√ F13Course subject matter should includeproblem based tasks√√ F14A consistent user interface should beprovided√ F15The development of a learningcommunity should be encouraged andnurtured√√√ F16Pre-course evaluation of learner profilesand learner needs should be carried out √ F17Tutors should prepare students to workcollaboratively√√ F18Social environments should be providedfor non-project communication√√ F19Course subject matter should encourageopinion diversity√√ F20An appropriate rationale for use ofComputer Mediated Communicationtechnologies should be developed√√ F21Technology used should enableasynchronous communication e.g. e-mail and bulletin boards√√√ Factor Factor DescriptionInitial List Validated Final F22Effective technical support should beprovided to both tutors and students √√ F23Group work should promote positiveinterdependence√√√ F24Students learning style should beconducive to sharing information withothers√ F25Group size should be kept small (e.g. 4-5 students)√ F26Course subject matter should bediscussion based√ F27 Tutors should be trained for their role √√√ F28Group processing discussions should beencouraged√√ F29Tutor should monitor groupcollaborative behaviour√ F30Technology used should enable multiplemeans of communication√√ F31Multiple tools are necessary, to appealto diverse learning styles, to supportcollaboration in multiple formats√ F32Technology should support multiplelearning styles√√ F33Course subject matter should bedelivered in multiple formats/strategiesincluding but not limited to discussionbased.√ F34At least one opportunity for face-to-faceinteraction with the group√ F35Tutor interaction with the group shouldbe limited so that they do not bias theCollaborative Learning√ F36Student’s interaction or participationshould be weighed such that they aregraded by their contribution to the group√ F37Collaborative tasks should promote thealignment of the individual goals ofstudents with the collective goals of thegroup√ F38Students should be encouraged tonegotiate task activities regularly toremain engaged.√ F39Group grades should be allocated forcollaborative effort√ F40Marks should be allocated for theprocess (or milestones during theprocess)√ F41 Software should be intuitive to use√ Factor Factor DescriptionInitial List Validated Final F42Technology should provide layers ofinteraction e.g. can go from browsing oruploading to being an intensive user√ F43Data should be collected and madeuseful e.g. tag clouds, trends√ F44Prior design of collaborative tasks isessential: i.e. design for learning, then e-moderate for participation√√ F45Instructional design of the activity,activity structure and assessment shouldpromote Collaborative Learning√√ F46Students should be assessed based ontheir collaboration and willingness tohelp others√ F47Students and tutors should be constantlyreminded of the (online) presence of theother members of the community√ F48There should be lots of opportunity forsocial communications in the early partof the course√√ F49Individual grades should be allocated tocollaborative tasks√ F50Learning environment should be userfriendly and kept simple√√ F51Group members should sign a groupcontract√ F52There should be personalized, detailed,quality-controlled feedback onassessment work√√ F53Students should have the opportunityonce or twice a year to meet with theirlecturers in a tutorial√ F54The development of teamwork skillsshould be explicitly built into theinstructional design√√